Installation¶

Just pip install:

pip install omegaconf

If you want to try this notebook after checking out the repository be sure to run python setup.py develop at the repository root before running this code.

Creating OmegaConf objects¶

Empty¶

In [1]:
xxxxxxxxxx
Outputs unchanged
{}
application/vnd.jupyter.stdout

From a dictionary¶

In [2]:
xxxxxxxxxx
Outputs unchanged
k: v
list:
- 1
- a: '1'
  b: '2'
application/vnd.jupyter.stdout

From a list¶

In [3]:
xxxxxxxxxx
Outputs unchanged
- 1
- a: 10
  b:
    a: 10
application/vnd.jupyter.stdout

From a yaml file¶

In [4]:
xxxxxxxxxx
Outputs unchanged
server:
  port: 80
log:
  file: ???
  rotation: 3600
users:
- user1
- user2
application/vnd.jupyter.stdout

From a yaml string¶

In [5]:
xxxxxxxxxx
Outputs unchanged
a: b
b: c
list:
- item1
- item2
application/vnd.jupyter.stdout

From a dot-list¶

In [6]:
xxxxxxxxxx
Outputs unchanged
a:
  aa:
    aaa: 1
    bbb: 2
  bb:
    aaa: 3
    bbb: 4
application/vnd.jupyter.stdout

From command line arguments¶

To parse the content of sys.arg:

In [7]:
xxxxxxxxxx
Outputs unchanged
server:
  port: 82
log:
  file: log2.txt
application/vnd.jupyter.stdout

Access and manipulation¶

Input yaml file:

In [8]:
xxxxxxxxxx
Outputs unchanged
server:
  port: 80
log:
  file: ???
  rotation: 3600
users:
- user1
- user2
application/vnd.jupyter.stdout

Object style access:¶

In [9]:
xxxxxxxxxx
Outputs unchanged
80
text/plain

dictionary style access¶

In [10]:
xxxxxxxxxx
Outputs unchanged
3600
text/plain

items in list¶

In [11]:
xxxxxxxxxx
Outputs unchanged
'user1'
text/plain

Changing existing keys¶

In [12]:
xxxxxxxxxx

Adding new keys¶

In [13]:
xxxxxxxxxx

Adding a new dictionary¶

In [14]:
xxxxxxxxxx

providing default values¶

In [15]:
xxxxxxxxxx
Outputs unchanged
'a default value'
text/plain

Accessing mandatory values¶

Accessing fields with the value ??? will cause a MissingMandatoryValue exception. Use this to indicate that the value must be set before accessing.

In [16]:
xxxxxxxxxx
Outputs unchanged
Missing mandatory value: log.file
    full_key: log.file
    object_type=dict
application/vnd.jupyter.stdout

Variable interpolation¶

OmegaConf support variable interpolation, Interpolations are evaluated lazily on access.

Config node interpolation¶

The interpolated variable can be the dot-path to another node in the configuration, and in that case the value will be the value of that node.

Interpolations are absolute by default. Relative interpolation are prefixed by one or more dots: The first dot denotes the level of the node itself and additional dots are going up the parent hierarchy. e.g. ${..foo} points to the foo sibling of the parent of the current node.

In [17]:
xxxxxxxxxx
Outputs unchanged
server:
  host: localhost
  port: 80
client:
  url: http://${server.host}:${server.port}/
  server_port: ${server.port}
  description: Client of ${.url}
application/vnd.jupyter.stdout
In [18]:
xxxxxxxxxx
Outputs unchanged
conf.client.server_port:  80 int
conf.client.url:  http://localhost:80/ str
application/vnd.jupyter.stdout

to_yaml() will resolve interpolations if resolve=True is passed

In [19]:
xxxxxxxxxx
Outputs unchanged
server:
  host: localhost
  port: 80
client:
  url: http://localhost:80/
  server_port: 80
  description: Client of http://localhost:80/
application/vnd.jupyter.stdout

Interpolations may be nested, enabling more advanced behavior like dynamically selecting a sub-config:

In [20]:
xxxxxxxxxx
Outputs unchanged
Default: cfg.plan = plan A
After selecting plan B: cfg.plan = plan B
application/vnd.jupyter.stdout

Interpolated nodes can be any node in the config, not just leaf nodes:

In [21]:
xxxxxxxxxx
Outputs unchanged
(180, 75)
text/plain
xxxxxxxxxx
3
 
1
## Environment variable interpolation
2
​
3
Environment variable interpolation is also supported.
⇛⇚
xxxxxxxxxx
4
 
1
## Environment variable interpolation
2
​
3
Environment variable interpolation is also supported.
4
An environment variable is always returned as a string.
In [22]:
In [22]:
xxxxxxxxxx
3
 
1
# Let's set up the environment first (only needed for this demonstration)
2
import os
3
os.environ['USER'] = 'omry'
⇛⇚
xxxxxxxxxx
4
 
1
# Let's set up the environment first (only needed for this demonstration)
2
import os
3
os.environ['USER'] = 'omry'
4
os.environ['USERID'] = '123456'
xxxxxxxxxx
1
 
1
Here is an example config file interpolates with the USER environment variable:
⇛⇚
xxxxxxxxxx
1
 
1
Here is an example config file interpolating with the USER and USERID environment variables:
In [23]:
In [23]:
xxxxxxxxxx
2
 
1
conf = OmegaConf.load('../source/env_interpolation.yaml')
2
print(OmegaConf.to_yaml(conf))
Outputs changed
user:
  name: ${env:USER}
  home: /home/${env:USER}


user:
  name: ${oc.env:USER}
  home: /home/${oc.env:USER}
  id: ${oc.env:USERID}
application/vnd.jupyter.stdout
In [24]:
In [24]:
xxxxxxxxxx
2
 
1
conf = OmegaConf.load('../source/env_interpolation.yaml')
2
print(OmegaConf.to_yaml(conf, resolve=True))
Outputs changed
user:
  name: omry
  home: /home/omry


user:
  name: omry
  home: /home/omry
  id: '123456'
application/vnd.jupyter.stdout
xxxxxxxxxx
1
 
1
You can specify a default value to use in case the environment variable is not defined. The following example sets `abc123` as the the default value when `DB_PASSWORD` is not defined.
⇛⇚
xxxxxxxxxx
4
 
1
You can specify a default value to use in case the environment variable is not defined.
2
This default value must be a string (with the exception of `${oc.env:VAR,null}` that returns `None` if `VAR` is not defined).
3
​
4
The following example sets `abc123` as the the default value when `DB_PASSWORD` is not defined:
In [25]:
In [25]:
xxxxxxxxxx
3
 
1
os.environ.pop('DB_PASSWORD', None)  # ensure env variable does not exist
2
cfg = OmegaConf.create({'database': {'password': '${env:DB_PASSWORD,abc123}'}})
3
cfg.database.password
⇛⇚
xxxxxxxxxx
3
 
1
os.environ.pop('DB_PASSWORD', None)  # ensure env variable does not exist
2
cfg = OmegaConf.create({'database': {'password': '${oc.env:DB_PASSWORD,abc123}'}})
3
cfg.database.password
Outputs unchanged
'abc123'
text/plain
xxxxxxxxxx
1
 
1
## Decoding strings with interpolations
Metadata changed
xxxxxxxxxx
xxxxxxxxxx
4
 
1
You can automatically convert a string to its corresponding type (e.g., bool, int, float, dict, list) using `oc.decode` (which can even resolve interpolations).
2
This resolver also accepts ``None`` as input, in which case it returns ``None``.
3
​
4
This can be useful for instance to parse environment variables:
Metadata changed
xxxxxxxxxx
In [26]:
xxxxxxxxxx
20
 
1
cfg = OmegaConf.create(
2
    {
3
        "database": {
4
            "port": '${oc.decode:${oc.env:DB_PORT}}',
5
            "nodes": '${oc.decode:${oc.env:DB_NODES,null}}',
6
            "timeout": '${oc.decode:${oc.env:DB_TIMEOUT,null}}',
7
        }
8
    }
9
)
10
​
11
os.environ["DB_PORT"] = "3308"  # integer
12
os.environ["DB_NODES"] = "[host1, host2, host3]"  # list
13
os.environ.pop("DB_TIMEOUT", None)  # unset variable
14
​
15
print("port (int):", repr(cfg.database.port))
16
print("nodes (list):", repr(cfg.database.nodes))
17
print("timeout (missing variable):", repr(cfg.database.timeout))
18
​
19
os.environ["DB_TIMEOUT"] = "${.port}"
20
print("timeout (interpolation):", repr(cfg.database.timeout))
Metadata changed
xxxxxxxxxx
port (int): 3308
nodes (list): ['host1', 'host2', 'host3']
timeout (missing variable): None
timeout (interpolation): 3308
application/vnd.jupyter.stdout
xxxxxxxxxx
1
 
1
Environment variables are parsed when they are recognized as valid quantities that may be evaluated (e.g., int, float, dict, list):
Metadata changed
xxxxxxxxxx
In [26]:
xxxxxxxxxx
12
 
1
cfg = OmegaConf.create({'database': {'password': '${env:DB_PASSWORD,abc123}',
2
                                    'user': 'someuser',
3
                                    'port': '${env:DB_PORT,3306}',
4
                                    'nodes': '${env:DB_NODES,[]}'}})
5
​
6
os.environ["DB_PORT"] = '3308'  # integer
7
os.environ["DB_NODES"] = '[host1, host2, host3]'  # list
8
os.environ["DB_PASSWORD"] = 'a%#@~{}$*&^?/<'  # string
9
​
10
print(repr(cfg.database.port))
11
print(repr(cfg.database.nodes))
12
print(repr(cfg.database.password))
Metadata changed
xxxxxxxxxx
3308
['host1', 'host2', 'host3']
'a%#@~{}$*&^?/<'
application/vnd.jupyter.stdout

Custom interpolations¶

You can add additional interpolation types using custom resolvers. The example below creates a resolver that adds 10 to the given value.

In [27]:
xxxxxxxxxx
Outputs unchanged
1000
text/plain

You can take advantage of nested interpolations to perform custom operations over variables:

In [28]:
xxxxxxxxxx
Outputs unchanged
3
text/plain

By default a custom resolver’s output is cached, so that when it is called with the same inputs we always return the same value. This behavior may be disabled by setting use_cache=False:

In [29]:
xxxxxxxxxx
Outputs unchanged
With cache: 7220 7220
Without cache: 1914 122
application/vnd.jupyter.stdout

Merging configurations¶

Merging configurations enables the creation of reusable configuration files for each logical component instead of a single config file for each variation of your task.

Machine learning experiment example:

conf = OmegaConf.merge(base_cfg, model_cfg, optimizer_cfg, dataset_cfg)

Web server configuration example:

conf = OmegaConf.merge(server_cfg, plugin1_cfg, site1_cfg, site2_cfg)

The following example creates two configs from files, and one from the cli. It then combines them into a single object. Note how the port changes to 82, and how the users lists are combined.

In [30]:
xxxxxxxxxx
Outputs unchanged
server:
  port: 80
users:
- user1
- user2
application/vnd.jupyter.stdout
In [31]:
xxxxxxxxxx
Outputs unchanged
log:
  file: log.txt
application/vnd.jupyter.stdout
In [32]:
xxxxxxxxxx
Outputs unchanged
server:
  port: 82
users:
- user1
- user2
log:
  file: log.txt
application/vnd.jupyter.stdout